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PROGRAM DEVELOPMENT THROUGH FORMATIVE EVALUATION: THE SWRL 

INSTRUCTIONAL CONCEPTS PROGRAM 

Roger 0. Scott 



ABSTRACT 

The SWRL Instructional Concepts Program has been developed to 
help kindergarten children quickly learn conceptual skills which are 
fundamental to academic achievement. Formative evaluation procedures 
employed in the 1968-69 tryout were primarily responsible for the 
changes incorporated into the present program. These procedures in- 
cluded teacher interviews, observations of lessons being taught, 
student tests for each unit of the program, and collection and ana- 
lysis of pretest and posttest performance data. It is expected that 
the revised Instructional Concepts Program will be more effective 
for students and more acceptable to teachers because of the revisions 
resulting from the formative evaluation procedures. 
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PROGRAM DEVELOPMENT THROUGH FORMATIVE EVALUATION: THE SWRL 

INSTRUCTIONAL CONCEPTS PROGRAM 



Recent literature in educational research and development has 
emphasized the use of formative evaluation procedures. Formative 
evaluation emphasizes the collection of data during, rather than after 
the development of an instructional program. The information obtained 
through this type of evaluation is used to make appropriate revisions 
in instructional materials and teaching procedures. 

Central to the development of instructional programs at the 
Southwest Regional Laboratory for Educational Research and Development 
(SWRL) is the use of systematic formative evaluation procedures . In 
the early stages of program development, individual components of the 
instructional package are tested in a small number of classrooms. 

Later, the entire program is tested in many classrooms that are repre- 
sentative of those for which the instruction is intended. Each of 
these tryouts is designed for the purpose of collecting information 
relevant to appropriate program revisions. The tryout/revision cycle 
is repeated until the program is proven effective. 

The application of formative evaluation to the initial preparation 
and improvement of an instructional program can be illustrated by the 
development of the SWRL Instructional Concepts Program. This is a one- 
semester program designed to teach kindergarten children to respond 
appropriately to 86 concept words . This paper describes the use of 
formative evaluation procedures in the initial tryout and first revision 
of the 1968-69 Instructional Concepts Program. 



Method 



Instructional Program 

The Instructional Concepts Program helps kindergarten children 
learn to respond appropriately to a large number of concept words. 
Included in these objectives are colors, sizes, positions, and amounts. 
The program is designed to teach children to name and identify examples 
of each concept. 

Instruction to teach the concepts was, in the initial version of 
the program, incorporated into 32 lessons, each lesson focusing on two 
to four concepts. Lessons consisted of a short story built around the 
concepts and large poster illustrations of the story. Materials for 
several optional activities were also provided. These included instruc- 
tional games, flashcards, and practice exercises for each student. Each 
teacher received a manual which contained the stories and games for all 
lessons and a description of the recommended teaching procedures. 
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The program was divided into five instructional units. Criterion 
exercises to be used by the teacher in assessing the concept skills of 
the children were prepared for each unit. 



Subjects 

More than 200 kindergarten children in three California schools 
took part in the 1968-69 tryout of the Instructional Concepts Program. 
Two schools were located in the inner city of a large urban district. 
The other school was located in a small community in an agricultural 
region. These tryout schools were selected on the basis of scores on 
a 10- item pretest which sampled the skills to be taught in the program. 
The test was individually administered to a small sample of students in 
each of five schools. Mean scores were below 50% correct for the 
three schools which were selected. The participation of two other 
schools was rejected because of higher scores. 

Most of the children in the three schools were from low-income 
families. Approximately one-half of the children in the tryout were 
Negro; nearly all of the other children were Mexican-American . 



School Liaison 

Before the tryout began, a teacher orientation meeting was held 
at SWRL. At the conclusion of the tryout, the teachers attended a 
day-long conference devoted to revisions of the program. Throughout 
the 15 week tryout, staff members made numerous phone calls and visits 
to the schools. In addition, the five unit tests which were administered 
by the teachers were returned to the Laboratory for scoring and 
summarizing. These results were then given to the teachers. 



Instructional Outcomes 

To determine what concepts children should know, Laboratory staff 
members reviewed several first-grade curriculum guides in current use. 
These materials make certain implicit assumptions concerning the concep- 
tual abilities of first graders. For instance, the curriculum guide 
which advises, "Tell the children to turn to the next page," implies 
that students will know concepts such as "turn," "next," and "page." 
Concepts which occurred in at least two of the curriculum guides and 
which did not refer to specific objects (e.g., house, dog, tree, etc.), 
were adopted for the program. This list of concepts was revised on 
the basis of suggestion's : from primary-grade teachers and curriculum 
specialists and individual assessment of kindergarten children on 
tasks involving use of the various concepts. The result was a list 
of 86 concepts grouped into seven categories: color, size, shape, 

position, amount, time, and equivalence. A list of the concepts in 
each category appears in Appendix A. 
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Criterion Tests 

Concept identification test . An identification test was constructed 
to measure children’s knowledge of the concepts. Five concepts were 
randomly selected from six of the program's concept categories. An 
additional item tested a time concept. 

One test item was constructed for each selected concept. An addi- 
tional five items tested the use of concepts in conjunction. For exam- 
ple, children were asked to identify a triangle under a line. This 
tested two concepts: under and triangle. Concepts tested in conjunction 

were randomly sampled from those already selected for the test. 

The required response in the test was the identification of the 
orally presented concept by pointing. The test had a total of 36 items, 
each consisting of a picture illustration of a concept and two dis- 
tractors. Examples of test items used in each category appear in 
Appendix B. 

The reliability coefficient for the individually administered 
instrument for 181 subjects taking the pretest was .75 (Kuder- 
Richardson 21) . 

Constructed response concept test . A concept naming test was 
constructed using 15 of the concepts and concept illustrations which 
were included in the identification test. Five color concepts, five 
shape concepts, and four amount concepts were tested, in addition 
to one item which measured the conjunctive use of concepts. Each 
child was asked questions such as "What color is this?" "What shape 
is this?" or "How many are there?" The reliability coefficient for 
the constructed response test was .55 (Kuder-Richardson 21) . 



Results and Discussion 



Concept Identification 

The children enrolled in the eight tryout classes made substantial 
improvements in their ability to identify concepts. Mean scores on the 
36-item concept identification test were 23.82 before the program and 
28.73 after the 15 weeks of instruction. These scores, when corrected 
for guessing, show a progression from 49% correct on the pretest to 
70% correct on the posttest. 

The corrected pretest and posttest concept identification scores 
for each class are presented in Table 1. There was considerable varia- 
tion between classes with the percentage increase ranging from 14 to 28. 
There was a greater variation when the concept identification scores 
for each concept category were compared. These figures are presented 
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TABLE 1 

MEAN PRETEST AND POSTTEST CLASS SCORES 
FOR THE CONCEPT IDENTIFICATION TEST 



CLASS 


NUMBER OF 
STUDENTS 


MEAN NUMBER CORRECT 


MEAN PERCENTAGE CORRECT 


PRETEST 


POSTTEST 


PRETEST 


POSTTEST 


1 


14 


22.29 


27.32 


62% 


76% 


o 


17 


19.77 


25.59 


55% 


717 


3 


12 


19.62 


27.12 


54% 


75% 


4 


15 


19.31 


25.40 


54% 


707 


5 


19 


16.82 


26.93 


47% 


75% 


6 


22 


16.10 


22.23 


45% 


62% 


7 


15 


15.11 


23.40 


42% 


65% 


8 


19 


15.00 


24.32 


42% 


67% 


TOTAL 


133 


17.73 


25.10 


49% 


70% 



NOTE: All of the scores presented above have been corrected for 

guessing. The test contained 32 three-choice items and four two- 
choice items. 
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in Table 2. Shape and position concepts showed the highest percentage 
gains — 30 and 28 respectively. The least gain was in size concepts 
where scores increased on 11% over the pretest. There was still more 
variation when posttest scores of the concept categories were compared. 
At one extreme, children were nearly perfect in their ability to 
identify color concepts. At the low end were equivalence concepts, 
with a score of slightly over 50%, and the time concept which had a 
low score because the single item measuring the concept apparently 
cued a wrong response, judging by the less than chance results on 
the pretest . 

The distribution of pretest and posttest scores is presented in 
Figure 1. An examination of these data shows that the shape of these 
distributions is significantly different (F=1.83, df =132, 132). The 
standard deviation was reduced from 5.81 on the pretest to 4.29 on the 
posttest. This is an important result, since it reflects the fact that 
many of the children who made very low scores on the pretest narrowed 
the gap between themselves and the rest of the class. 



Concept Naming 

The concept naming skills were assessed after the 15 weeks of 
instruction. The scores were surprisingly high relative to the selected 
response scores of the concept identification test. Of a possible 15 
correct, the mean was 12.60. The overall percentage score was 84. 

Because this test asked children to give the concept name of individ- 
ually presented stimulus objects, there was no need to correct scores 
for guessing as in the case of the multiple-choice concept identifica- 
tion test. The mean scores for each concept category are presented in 
Table 3. The concept identification test scores for the same concept 
categories are also listed. 

The data in Table 3 indicate that naming skills are either equal 
to or higher than identification skills. The reverse had been expected. 
Children might be able to point to an exemplar of a concept but not 
name it. The results are especially surprising since many of the children 
used Spanish as their first language. One possible answer involves the 
validity of the concept identification test. It is possible that some 
children who could identify a given amount concept did not do so on the 
identification test because of the counting required when the concept 
was presented with two distractors. A child might have to count three 
sets of objects before finding the right choice. In the naming test, 
the concept exemplar was presented without the distractors. Thus, a 
child could arrive at the answer with only one counting operation. 



Control Group 

Some of the increase in the conceptual skills of kindergarten 
children can be attributed to the traditional school program and 



ERLC 



8 



-7- 



TABLE 2 

MEAN PRETEST AND POSTTEST CONCEPT CATEGORY SCORES 
FOR THE CONCEPT IDENTIFICATION TEST 



CONCEPT CATEGORY MEAN NUMBER CORRECT MEAN PERCENTAGE CORRECT 





w 

PRETEST 

• 


POSTTEST 


PRETEST 


POSTTEST 


Color 


3.98 


4.78 


80% 


96% 


Size 


3.26 


3.80 


65% 


76% 


Conjunctive Concepts 


2.75 


3.98 


55% 


80% 


Amount 


2.41 


3.20 


48% 


64% 


Shape 


1.84 


3.34 


37% 


67% 


Equivalence 


1.82 


2.71 


36% 


54% 


Position 


1.75 


3.14 


35% 


63% 


Time 3 


Less Than 


.15 


Less Than 


15% 




Chance 




Chance 




Total 


17.73 


25.10 


49% 


70% 


NOTE: All of the scores 


presented above have 


been corrected 


for 



guessing . 

a The Test contained only one item for the "Time" Category and five items 
each for the other seven categories . 
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Figure 1. Pretest and posttest distributions for the concept 
identification test. 



Number of 
Children 



Number of 
Children 



14 
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TABLE 3 

MEAN CONCEPT CATEGORY SCORES 
FOR NAMING AND POINTING TASKS 



CONCEPT CATEGORY CONCEPT NAMING TEST CONCEPT IDENTIFICATION 

TEST (POINTING TASK) 



MEAN NUMBER MEAN PERCENTAGE MEAN PERCENTAGE CORRECT 
CORRECT CORRECT 



Color 3 


4.84 


97 


96 


Shape 3 


3.95 


79 


67 


Amount^ 


3.02 


76 


64 


Conjunctive 


.79 


79 


80 


Concepts 3 








TOTAL 


12.60 


84 


79 



NOTE: Concept naming test scores not corrected for guessing. 

a Five test items for naming 
^Four test items for naming 
c One test item for naming 
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extra-school experiences. To investigate the importance of these 
factors, the concept identification test was administered to 80 kin- 
dergarten children who were not involved in the tryout. The test was 
given to children from two inner city San Diego schools at the end of 
the school year. The children in these schools are from families with 
income levels similar to those in the tryout schools . The data from 
this test administration are reported in Table 4. 

After instruction, the tryout children scored much higher than the 
comparison group. The scores show that the children who did not have 
access to the SWRL program were, at the end of the year, still close to 
beginning-of-the-year skills levels. It seems likely that the SWRL 
program enabled teachers to be more effective in teaching basic instruc- 
tional concepts. 



Program Revisions 



A major purpose of the 1968-69 tryout was to increase the effective- 
ness of the program by identifying and incorporating improvements in the 
instructional materials and procedures. Formative evaluation data were 
of primary importance in making these revisions. These data included 
reports of participating teachers, classroom observations of SWRL staff, 
analyses of criterion exercise data, and suggestions of curriculum spe- 
cialists from the participating districts. 



Instructional Outcomes 

Most of the concepts taught in the original Instructional Concepts 
Program are included in the revised program. Four concepts relating to 
prereading skills were added, since the program is used before children 
receive any reading instruction. The concept "not" was added at the 
request of teachers and curriculum specialists in the tryout schools. 
Although teachers liked the lessons dealing with pattern they agreed 
that these concepts were not critical for future academic performance 
and consequently they were dropped from the objectives. During the 
tryout many children had difficulty completing the group administered 
criterion exercises. For this reason ten concepts dealing with direc- 
tion following skills were added to the program. These concepts include 
page, row, turn, mark and oval, and recognition of numerals 1, 2, 3, 4 
and 5. 



Sequence and Structure 

The revised program is divided into seven units . Unlike the 
original version, each unit contains concepts related to a single 
dimension such as color, size or amount. This was done at the 
request of tryout teachers who felt that such an arrangement would 
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TABLE 4 

MEAN CONCEPT CATEGORY SCORES 
FOR THE CONCEPT IDENTIFICATION TEST 



CONCEPT CATEGORY 


CHILDREN WITH 
N = 


SWRL MATERIALS 
133 


COMPARISON GROUP 
N = 80 




PRETEST 


POSTTEST 


POSTTEST 


Color 


3.98 


4.78 


4.10 


Size 


3.26 


3.80 


3.11 


Conjunctive Concepts 


2.75 


3.98 


2.13 


Shape 


1.84 


3.34 


2.68 


Position 


1.75 


3.14 


1.18 


TOTAL 


13.58 


19.04 


13.20 



NOTE: All of the scores presented above have been corrected for 

guessing. 

Only the portions of the concept identification test which were 
identical in the administration to both groups are reported. 
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facilitate evaluation of student performance and scheduling of addi- 
tional practice. The units were sequenced according to pretest data. 
Scores were highest on colors, so that unit was scheduled first; the 
next highest scores were on sizes, so the unit on sizes was sequenced 
second, and so on. 



Instructional Components 

The revised Instructional Concepts Program consists of 10 major 
components. Each component was revised on the basis of formative 
evaluation data. 

Early in the tryout , it was determined that the format of the 
story illustrations would have to be changed. One poster was used 
to illustrate each of the stories in the revised program. Three cards 
were used to illustrate each story in the original program, but teachers 
reported that the posters were cumbersome. Lesson observations by SWRL 
staff also indicated that the posters were used in a manner which pre- 
vented children from frequently practicing the use of the concepts in 
the lesson. Teachers typically asked individual children to come to 
the front of the room and point to an instance of the concept illustrated 
on the poster. With a lesson conducted in this manner, many children 
did not have a chance to engage in appropriate practice. Others had 
only a very limited opportunity. In order to increase the frequency 
of practice, concept books were developed for the revised program. All 
children received a book for each of the program's seven units. These 
books are similar in format to the storybooks used in the SWRL Communi- 
cation Skills Program. Each lesson is illustrated on two pages which 
face each other. The illustrations include the unit theme character 
and objects familiar to inner city kindergarten children. The illus- 
trations also represent two or more instances of each concept included 
in that lesson. Concept naming and identifying questions to ask the 
class are listed in each book. 

Practice exercises were also found in need of major revisions. In 
the original program a single page which illustrated the concepts to 
be taught was included for each lesson. This was an optional activity 
which teachers could hand out and ask children to color or mark. A 
number of teachers suggested that this component was not structured 
enough to be useful in the class. Because of these comments and be- 
cause of a desire to coordinate the program with the SWRL Communication 
Skills materials, the practice exercises were completely revised. Each 
revised exercise consists of four pages with each page divided into 
five rows. Directions are printed in the margin of each row so that 
they can be read from the left hand side of the paper. These directions . 
which can be used by the teacher, an aide, a parent, or a tutor ask the 
child to identify illustrated concepts by pointing and naming. 
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Other program components which underwent major format changes as a 
result of the tryout included games and the teacher's manual. Lesser 
changes were made for the flashcards, criterion exercises, training 
lessons for the criterion exercises, and the pre- and posttest. One 
component — concept cards — was added to the program at the request of 
several tryout teachers. 
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APPENDIX A 

CONCEPTS AND CONCEPT CATEGORIES 
OF THE ORIGINAL AND REVISED INSTRUCTIONAL CONCEPTS PROGRAM 



COLORS SHAPES 



Original 


Revised 


Original 


Revised 


red 


red 


circle 


circle 


blue 


blue 


square 


square 


yellow 


yellow 


triangle 


triangle 


brown 


brown 


rectangle 


rectangle 


black 


black 


straight-line 


straight-line 


white 


white 


curved line 


curved line 


orange 


orange 






purple 


purple 






green 


green 








pink 







SIZES 

Original 


Revised 


Original 


AMOUNTS 

Revised 


larger 


larger - 


1 


1 


smaller 


smaller 


2 


2 


shorter 


shorter 


3 


3 


longer 


longer 


4 


4 


taller 


taller 


5 


5 


largest 


largest 


6 


6 


smallest 


smallest 


7 


7 


medium-sized 




8 


8 


shortest 


shortest 


9 


9 


tallest 


tallest 


10 


10 


longest 


longest 


least 


least 


thin 




most 


most 


thick 




more 


more 


thicker 


thicker 


less 


less 


thinner 


thinner 


all 


all 




thickest 


some 


some 




thinnest 


none 


none 






no 


no 
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POSITIONS COMPARISONS 



Original 


Revised 


Original 


Revised 


above 


above 


same 


same 


below 


below 


different 


different 


between 


between 


equal 


equal 


over 


over 


not equal 




under 


under 




unequal 


beside 


beside 




not 


outside 


outside 






behind 


behind 






in front of 




PATTERNS 




next to 


next to 






around 


around 


Original 


Revised 


top 


top 






bottom 


bottom 


striped 


(none) 


side 


side 


polka dotted 




front 


front 


checked 




middle 


middle 






end 


end 






1st 




PREREADING 


2nd 


2nd 


SKILLS 




3rd 


3rd 






4th 


4 th 


Original 


Revised 


back 


back 






beginning 


beginning 


(none) 


sound 


first 


first 




name 


last 


last 




letter 




in front 




word 




inside 








right 








left 


TIME 








Original 


Revised 






before 


before 






after 


after 








first 








last 
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APPENDIX B 

EXAMPLES OF TEST ITEMS IN EACH CONCEPT CATEGORY 
IN THE 1968-1969 INSTRUCTIONAL CONCEPTS PROGRAM 



1. COLOR 

"Point to the green bird." 

2. SHAPE 

"Point to the circle." 

3. SIZE 



"Point to the largest turtle." 

4 . AMOUNT 



"Point to the bowl with the most ice cream." 

5. POSITION 

"Point to the monkey at the beginning of the line." 

6. EQUIVALENCE 

"See this cat?" (Point to the cat in the margin.) "Point to 
a cat over here that is the same color." 

7 . CONJUNCTIVE 



"Point to the triangle under the line." 

8. TIME 



(Point to each picture in turn and tell the following story.) 
"Here is a boy sleeping in bed in the morning. Now he wakes 
up. Here he is going to school. Point to the picture that 
shows the boy before he woke up." 
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